Papers with spoken dialogue systems
Deep Learning for Conversational AI (N18-6)
Copied to clipboard
| Challenge: | Spoken Dialogue Systems (SDS) have great commercial potential . the advent of deep learning has led to significant advances in this area of NLP research . |
| Approach: | This tutorial will introduce researchers to the pipeline framework for modelling goal-oriented dialogue systems. |
| Outcome: | This tutorial will familiarise researchers with the latest advances in spoken dialogue systems . the aim of the course is to encourage dialogue research in the NLP community . |
Natural Language Generation by Hierarchical Decoding with Linguistic Patterns (N18-2)
Copied to clipboard
| Challenge: | a common and mostly adopted method is the rule-based (or template-based) method for natural language generation. |
| Approach: | They propose a hierarchical decoding NLG model based on linguistic patterns in different levels. |
| Outcome: | The proposed method outperforms the traditional one with a smaller model size. |
Practical Application of Domain Dependent Confidence Measurement for Spoken Language Understanding Systems (N18-3)
Copied to clipboard
| Challenge: | a confidence score is a scalar quantity that measures the reliability of an automatic system. |
| Approach: | They propose to use a confidence measure to evaluate the reliability of an SLU system . they build confidence models for three different types of dialogue states . |
| Outcome: | The proposed model can be used to reject low-confidence SLU results in real-world scenarios. |
PyOpenDial: A Python-based Domain-Independent Toolkit for Developing Spoken Dialogue Systems with Probabilistic Rules (D19-3)
Copied to clipboard
| Challenge: | a recent development of spoken dialogue systems has enabled deep learning to achieve state-of-the-art performance. |
| Approach: | They propose a Python-based domain-independent, open-source toolkit for spoken dialogue systems. |
| Outcome: | The proposed toolkit extends OpenDial's Java-based architecture and provides new functions for neural dialogue state tracking and action planning. |
DART: Open-Domain Structured Data Record to Text Generation (2021.naacl-main)
Copied to clipboard
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, Nazneen Fatema Rajani
| Challenge: | Data-to-text annotations can be costly when dealing with tables with nontrivial structures. |
| Approach: | They propose a procedure for extracting semantic triples from tables that encodes their structures by exploiting table headers and table title. |
| Outcome: | The proposed method exploits the semantic dependencies between table headers and title to extract semantic triples from tables. |
Actor-Double-Critic: Incorporating Model-Based Critic for Task-Oriented Dialogue Systems (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In order to improve the sample-efficiency of deep reinforcement learning, we implemented imagination augmented agent (I2A) in spoken dialogue systems (SDS). |
| Approach: | They propose to use an actor-double-critic to improve the stability and overall performance of imagination augmented agent (I2A) in spoken dialogue systems. |
| Outcome: | The proposed model-based agent (ADC) improves the stability and sample-efficiency of deep reinforcement learning (DRL) on a restaurant booking task. |
End-to-end ASR to jointly predict transcriptions and linguistic annotations (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models generate audio transcripts by sequentially producing likely graphemes, or multi-graphemic units, from which lexical items of a language can be recovered. |
| Approach: | They propose a Transformer-based sequence-to-sequence model for automatic speech recognition that can produce high-quality transcriptions and linguistic annotations. |
| Outcome: | The proposed model can produce high-quality transcriptions and linguistic annotations on Japanese and English audio datasets. |
PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that process both text and speech face problems in response generation latency. |
| Approach: | They propose to extend the input and output sequences of the language model to support the parallel generation of text and speech. |
| Outcome: | The proposed model improves latency while maintaining quality of response content while maintaining the quality of the response content. |
Investigating the Impact of Incremental Processing and Voice Activity Projection on Spoken Dialogue Systems (2025.coling-main)
Copied to clipboard
| Challenge: | a large language model (LLM) is gaining attention for its ability to model human-like turn-taking in human conversations. |
| Approach: | They developed a turn-taking model that can be trained in unsupervised manner using spoken dialogue data between two speakers. |
| Outcome: | The proposed model can be trained in unsupervised manner using spoken dialogue data between two speakers. |
Multi-task and Multi-lingual Joint Learning of Neural Lexical Utterance Classification based on Partially-shared Modeling (C18-1)
Copied to clipboard
| Challenge: | Existing studies on multitask and multilingual joint learning focus on cross-task or cross-lingual knowledge transfer. |
| Approach: | They propose to divide state-of-the-art neural lexical utterance classification into language-specific components that can be shared between different tasks and different languages. |
| Outcome: | The proposed method is able to support multi-task and multi-lingual learning using Japanese and English data sets with three different lexical utterance classification tasks. |
A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding (D18-1)
Copied to clipboard
| Challenge: | Spoken language understanding (SLU) involves intent determination and slot filling . existing joint learning methods only consider joint learning by sharing parameters on surface level rather than semantic level. |
| Approach: | They propose a self-attentive model to fully utilize the semantic correlation between slot and intent. |
| Outcome: | The proposed model outperforms existing methods in both intent detection and slot filling tasks on ATIS benchmarks. |
Dialogue Systems Can Generate Appropriate Responses without the Use of Question Marks?– a Study of the Effects of “?” for Spoken Dialogue Systems – (2024.lrec-main)
Copied to clipboard
Tomoya Mizumoto, Takato Yamazaki, Katsumasa Yoshikawa, Masaya Ohagi, Toshiki Kawamoto, Toshinori Sato
| Challenge: | Existing systems for spoken dialogues do not append question marks to recognized queries . lack of punctuation marks in spoken dialogue can negatively impact comprehension . |
| Approach: | They investigate the impact of question marks on spoken dialogue systems . they analyze examples to determine which types of utterances have the impact . |
| Outcome: | The proposed method shows that question marks have a significant impact on spoken dialogue systems. |
FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent research has made significant progress in developing empathetic spoken chatbots based on large language models (LLMs). |
| Approach: | They propose an end-to-end empathetic spoken chatbot trained efficiently that generates emotionally expressive speech and outperforms other emmpathetic models in emphatic dialogue, SER, and SpokenQA tasks. |
| Outcome: | The proposed model outperforms other empathetic models on e-dialog, SER, and SpokenQA tasks and achieves strong results on several speech tasks. |
PACHAT: Persona-Aware Speech Assistant for Multi-party Dialogue (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive research on spoken dialogue systems has advanced the development of intelligent voice assistants, but integration of role information within speech remains an underexplored area. |
| Approach: | They propose a language-based spoken dialogue system that integrates role information within speech to generate contextually appropriate responses. |
| Outcome: | The proposed architecture achieves speaker-specific responses, character understanding, and the generation of targeted replies in multi-party dialogue scenarios, surpassing existing spoken dialogue systems. |
Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction (2026.acl-long)
Copied to clipboard
Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong He
| Challenge: | End-to-end (E2E) spoken dialogue systems are replacing cascaded pipelines for voice-based human-AI interaction. Existing benchmarks evaluate these systems on synthetic speech and single-turn tasks, leaving multi-turn conversational ability underexplored. |
| Approach: | They propose an open-source benchmark to evaluate spoken dialogue systems under natural multi-turn interaction patterns. |
| Outcome: | The proposed model fails on the highest-performing model with 54.65% pass rate. |
Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback (2026.findings-acl)
Copied to clipboard
Siddhant Arora, Jinchuan Tian, Jiatong Shi, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe
| Challenge: | Existing studies on reinforcement learning from human or AI feedback have focused on semantic rewards at the utterance level. |
| Approach: | They propose a multi-reward RLAIF framework for speech-in/speech-out dialogue systems . they combine semantic, audio-quality, and emotion-consistency rewards . |
| Outcome: | The proposed framework improves speech-in/speech-out dialogue system quality . it combines semantic, audio-quality, and emotion-consistency rewards . the proposed framework is available to download from the cdc. |